Fixed rate limit issue in bulk publish #2022

sunil-lakshman · 2025-07-28T07:55:52Z

Fixed rate limit issue in bulk publish.

Smart Rate Limiter Implementation
✅ Created SmartRateLimiter class with intelligent rate limiting logic
✅ Implemented singleton pattern using Map to ensure one instance per organization
✅ Added proactive rate limiting to prevent 429 errors before they occur
2. Rate Limit State Management
✅ Added serverRateLimit tracking to store actual server limit from x-ratelimit-limit header
✅ Improved xRateLimitRemaining updates from server responses
✅ Added request counter to track total requests made
✅ Implemented lastStatusLog tracking to prevent duplicate status logs
3. Proactive Rate Limiting Features
✅ Low rate limit detection: Forces delays when xRateLimitRemaining <= 2
✅ Conservative batch sizing:
1 item at a time when rate limit ≤ 2
Max 2 items when rate limit ≤ 5
✅ Intelligent waiting: Prevents hitting zero rate limit
4. Queue Management
✅ Smart re-enqueuing: Items are re-added to queue after rate limit delays
✅ Error handling: 429 errors trigger delays and re-enqueuing
✅ Preservation of items: No data loss during rate limiting
5. Batch Size Optimization
✅ Dynamic batch sizing: Adapts to current rate limit
✅ Conservative approach: Smaller batches when rate limit is low
✅ Optimal processing: Balances speed with rate limit compliance
6. Error Prevention
✅ Prevents 429 errors: Proactive delays when rate limit is low
✅ Graceful degradation: Continues processing even with rate limits
✅ Recovery mechanism: Automatically resumes after delays

aman19K · 2025-07-28T10:22:38Z

packages/contentstack-bulk-publish/src/producer/unpublish.js

+async function processPendingUnpublishItems(stack, environment, locale, apiVersion) {
+  // Process assets first
+  while (pendingAssetItems.length > 0) {
+    const optimalBatchSize = smartRateLimiter.getOptimalBatchSize(pendingAssetItems.length);
+
+    if (optimalBatchSize === 0) {
+      // Rate limit exhausted, wait and retry
+      smartRateLimiter.logStatus();
+      await delay(1000);
+      continue;
+    }
+
+    // Take the optimal batch size
+    const batch = pendingAssetItems.splice(0, optimalBatchSize);
+
+    try {
+      await queue.Enqueue({
+        assets: batch,
+        Type: 'asset',
+        locale: locale,
+        environments: [environment],
+        stack: stack,
+        apiVersion,
+      });
+
+      smartRateLimiter.logStatus();
+
+    } catch (error) {
+      if (error.errorCode === 429) {
+        // Rate limit error, put items back and wait
+        pendingAssetItems.unshift(...batch);
+        smartRateLimiter.logStatus();
+        await delay(1000);
+      } else {
+        // Other error, log and continue
+        console.log(`Error processing asset batch: ${error.message}`);
+      }
+    }
+  }


@sunil-lakshman ,

Copilot

Pull Request Overview

This PR implements a comprehensive smart rate limiting system for the bulk publish functionality to prevent 429 rate limit errors and improve API request management. The solution introduces proactive rate limiting with intelligent batch sizing and queue management.

Implements SmartRateLimiter class with singleton pattern per organization for intelligent rate limiting
Replaces immediate bulk processing with pending item queues and optimal batch sizing
Adds comprehensive rate limit tracking and proactive delay mechanisms to prevent 429 errors

Reviewed Changes

Copilot reviewed 14 out of 16 changed files in this pull request and generated 4 comments.

Show a summary per file

File	Description
packages/contentstack/package.json	Updates bulk-publish package version to 1.9.1
packages/contentstack-utilities/src/contentstack-management-sdk.ts	Adds support for includeResHeaders in request headers
packages/contentstack-bulk-publish/src/util/smart-rate-limiter.js	New SmartRateLimiter class implementing intelligent rate limiting logic
packages/contentstack-bulk-publish/src/util/common-utility.js	Adds handleRateLimit utility function for rate limit management
packages/contentstack-bulk-publish/src/util/client.js	Enables response headers inclusion for rate limit tracking
packages/contentstack-bulk-publish/src/producer/*.js	Integrates smart rate limiter into publish/unpublish producers
packages/contentstack-bulk-publish/src/consumer/publish.js	Implements rate limit checking in all publish/unpublish operations

Files not reviewed (1)

pnpm-lock.yaml: Language not supported

Copilot · 2025-07-28T10:25:49Z

packages/contentstack-bulk-publish/src/consumer/publish.js

        delete assetobj.stack;
-        console.log(`Asset unpublished with Asset uid=${assetobj.assetUid}`);
+        console.log(chalk.red(`Could not Unpublish because of error=${formatError(error)}`));


This success handler is logging an error message with chalk.red instead of a success message with chalk.green. This appears to be copied from an error handler and should log a success message for asset unpublishing.

Suggested change

console.log(chalk.red(`Could not Unpublish because of error=${formatError(error)}`));

console.log(chalk.green(`Successfully unpublished asset with UID=${assetobj.assetUid} from environments=${assetobj.environments.join(', ')}`));

Copilot · 2025-07-28T10:25:49Z

packages/contentstack-bulk-publish/src/consumer/publish.js

        addLogs(
          logger,
          { options: assetobj, api_key: stack.stackHeaders.api_key, alias: stack.alias, host: stack.host },
-          'info',
+          'error',


This success handler is logging with 'error' level instead of 'info' level. This should be 'info' to match the success case.

Suggested change

'error',

'info',

Copilot · 2025-07-28T10:25:49Z

packages/contentstack-bulk-publish/src/consumer/publish.js

-          `Asset UID '${asset.uid}' ${asset.version ? `and version '${asset.version}'` : ''} ${
-            asset.locale ? `in locale '${asset.locale}'` : ''
-          }`,
+          `Asset UID '${asset.uid}' and version ${versionText} in locale ${localeText}`,


The string interpolation includes 'and version ${versionText}' but versionText already contains 'and version' prefix when present, causing duplication like 'and version and version 1'. Should be just Asset UID '${asset.uid}' ${versionText} ${localeText}.

Suggested change

`Asset UID '${asset.uid}' and version ${versionText} in locale ${localeText}`,

`Asset UID '${asset.uid}' ${versionText} ${localeText}`,

Copilot · 2025-07-28T10:25:50Z

packages/contentstack-bulk-publish/src/consumer/publish.js

@@ -350,24 +496,40 @@ async function performBulkPublish(data, _config, queue) {
      if (bulkPublishObj.apiVersion) {
        if (!isNaN(bulkPublishObj.apiVersion) && bulkPublishObj.apiVersion === apiVersionForNRP) {
          payload['api_version'] = bulkPublishObj.apiVersion;


This line is setting publish_with_reference on payload.details, but there's no guarantee that payload.details exists at this point. This could cause a runtime error if payload.details is undefined.

Suggested change

payload['api_version'] = bulkPublishObj.apiVersion;

payload['api_version'] = bulkPublishObj.apiVersion;

payload.details = payload.details || {};

aman19K · 2025-07-28T10:27:34Z

packages/contentstack-bulk-publish/src/producer/unpublish.js

+async function processPendingUnpublishItems(stack, environment, locale, apiVersion) {
+  // Process assets first
+  while (pendingAssetItems.length > 0) {
+    const optimalBatchSize = smartRateLimiter.getOptimalBatchSize(pendingAssetItems.length);
+
+    if (optimalBatchSize === 0) {
+      // Rate limit exhausted, wait and retry
+      smartRateLimiter.logStatus();
+      await delay(1000);
+      continue;
+    }
+
+    // Take the optimal batch size
+    const batch = pendingAssetItems.splice(0, optimalBatchSize);
+
+    try {
+      await queue.Enqueue({
+        assets: batch,
+        Type: 'asset',
+        locale: locale,
+        environments: [environment],
+        stack: stack,
+        apiVersion,
+      });
+
+      smartRateLimiter.logStatus();
+
+    } catch (error) {
+      if (error.errorCode === 429) {
+        // Rate limit error, put items back and wait
+        pendingAssetItems.unshift(...batch);
+        smartRateLimiter.logStatus();
+        await delay(1000);
+      } else {
+        // Other error, log and continue
+        console.log(`Error processing asset batch: ${error.message}`);
+      }
+    }
+  }
+
+  // Process entries
+  while (pendingEntryItems.length > 0) {
+    const optimalBatchSize = smartRateLimiter.getOptimalBatchSize(pendingEntryItems.length);
+
+    if (optimalBatchSize === 0) {
+      // Rate limit exhausted, wait and retry
+      smartRateLimiter.logStatus();
+      await delay(1000);
+      continue;
+    }
+
+    // Take the optimal batch size
+    const batch = pendingEntryItems.splice(0, optimalBatchSize);
+
+    try {
+      await queue.Enqueue({
+        entries: batch,
+        locale: locale,
+        Type: 'entry',
+        environments: [environment],
+        stack: stack,
+        apiVersion,
+      });
+
+      smartRateLimiter.logStatus();
+
+    } catch (error) {
+      if (error.errorCode === 429) {
+        // Rate limit error, put items back and wait
+        pendingEntryItems.unshift(...batch);
+        smartRateLimiter.logStatus();
+        await delay(1000);
+      } else {
+        // Other error, log and continue
+        console.log(`Error processing entry batch: ${error.message}`);
+      }
+    }
+  }
+}
+


@sunil-lakshman ,
Above code may lead to an infinite loop:-

If the API continuously responds with HTTP 429 errors and the smartRateLimiter.getOptimalBatchSize() keeps returning 0, the loop will wait 1 second and retry with optimalBatchSize === 0 and repeat forever.

Same situation if items are again added via unshift and never allowed to dequeue again due to rate limiting.

To Mitigate Infinite Loop you can use these approaches:- maximum retry count & Add exponential backoff instead of fixed 1 sec of delay.

aman19K · 2025-07-28T10:35:08Z

packages/contentstack-bulk-publish/src/util/common-utility.js

+ */
+async function handleRateLimit(error, data, delay, xRateLimitRemaining) {
+  // Check if rate limit is exhausted or batch size exceeds remaining limit
+  if (xRateLimitRemaining === 0 || data.length > xRateLimitRemaining) {


@sunil-lakshman , try to use optional chaining because we don't know the type of data property. Also what if xRateLimitRemaining will be undefined ?

aman19K · 2025-07-28T10:35:37Z

packages/contentstack-bulk-publish/src/util/smart-rate-limiter.js

+    // cliux.print(
+    //   `[${this.requestCount}] ${operation.toUpperCase()} ${itemType}: ${itemId} (${this.xRateLimitRemaining} remaining)`,
+    //   { color: 'cyan' }
+    // );


@sunil-lakshman , remove commented code.

aman19K · 2025-07-28T10:35:46Z

packages/contentstack-bulk-publish/src/util/smart-rate-limiter.js

+    // cliux.print(
+    //   `[${this.requestCount}] ✓ ${operation.toUpperCase()} ${itemType}: ${itemId} - SUCCESS`,
+    //   { color: 'green' }
+    // );


@sunil-lakshman , remove commented code.

aman19K · 2025-07-28T10:35:54Z

packages/contentstack-bulk-publish/src/util/smart-rate-limiter.js

+    // cliux.print(
+    //   `[${this.requestCount}] ✗ ${operation.toUpperCase()} ${itemType}: ${itemId} - FAILED (${errorCode})`,
+    //   { color: 'red' }
+    // );


@sunil-lakshman , remove commented code.

aman19K · 2025-07-28T10:36:31Z

packages/contentstack-bulk-publish/src/util/smart-rate-limiter.js

+    const errorCode = error.errorCode || 'UNKNOWN';
+    const errorMessage = error.message || error.error_message || 'Unknown error';


@sunil-lakshman , try to use optional chaining as we aren't sure about type of error.

netrajpatel · 2025-07-28T10:54:13Z

packages/contentstack/README.md

@@ -3817,8 +3820,7 @@ USAGE
  $ csdx launch:functions [-p <value>] [-d <value>]

 FLAGS
-  -d, --data-dir=<value>  [default: /Users/aman.kumar/Documents/cli-repos/logger-v2/cli/packages/contentstack] Current
-                          working directory
+  -d, --data-dir=<value>  [default: /Users/sunil.lakshman/Documents/cli/packages/contentstack] Current working directory


@sunil-lakshman Do not use your local path; use '/' instead.

netrajpatel · 2025-07-28T10:55:43Z

packages/contentstack-bulk-publish/package.json

@@ -1,7 +1,7 @@
 {
  "name": "@contentstack/cli-cm-bulk-publish",
  "description": "Contentstack CLI plugin for bulk publish actions",
-  "version": "1.9.0",
+  "version": "1.9.1",


@sunil-lakshman This is not a small fix, consider bumping this to a minor version instead a patch version bump.

Fixed rate limit issue in bulk publish

e595752

aman19K reviewed Jul 28, 2025

View reviewed changes

sunil-lakshman requested a review from Copilot July 28, 2025 10:24

Copilot AI reviewed Jul 28, 2025

View reviewed changes

aman19K requested changes Jul 28, 2025

View reviewed changes

netrajpatel requested changes Jul 28, 2025

View reviewed changes

sunil-lakshman closed this Aug 19, 2025

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Fixed rate limit issue in bulk publish #2022

Fixed rate limit issue in bulk publish #2022

Uh oh!

sunil-lakshman commented Jul 28, 2025 •

edited

Loading

Uh oh!

aman19K Jul 28, 2025

Uh oh!

Copilot AI left a comment

Uh oh!

Copilot AI Jul 28, 2025

Uh oh!

Copilot AI Jul 28, 2025

Uh oh!

Copilot AI Jul 28, 2025

Uh oh!

Copilot AI Jul 28, 2025

Uh oh!

aman19K Jul 28, 2025

Uh oh!

aman19K Jul 28, 2025

Uh oh!

aman19K Jul 28, 2025

Uh oh!

aman19K Jul 28, 2025

Uh oh!

aman19K Jul 28, 2025

Uh oh!

aman19K Jul 28, 2025

Uh oh!

netrajpatel Jul 28, 2025

Uh oh!

netrajpatel Jul 28, 2025

Uh oh!

Uh oh!

	console.log(chalk.red(`Could not Unpublish because of error=${formatError(error)}`));
	console.log(chalk.green(`Successfully unpublished asset with UID=${assetobj.assetUid} from environments=${assetobj.environments.join(', ')}`));

	`Asset UID '${asset.uid}' and version ${versionText} in locale ${localeText}`,
	`Asset UID '${asset.uid}' ${versionText} ${localeText}`,

	payload['api_version'] = bulkPublishObj.apiVersion;
	payload['api_version'] = bulkPublishObj.apiVersion;
	payload.details = payload.details \|\| {};

		const errorCode = error.errorCode \|\| 'UNKNOWN';
		const errorMessage = error.message \|\| error.error_message \|\| 'Unknown error';

Fixed rate limit issue in bulk publish #2022

Fixed rate limit issue in bulk publish #2022

Uh oh!

Conversation

sunil-lakshman commented Jul 28, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull Request Overview

Reviewed Changes

Uh oh!

Copilot AI Jul 28, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Jul 28, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Jul 28, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Jul 28, 2025

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

sunil-lakshman commented Jul 28, 2025 •

edited

Loading